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ABSTRACT 

Performance  of  military  operators  depends  on  both  physical  and  cognitive  aspects.  Enhancement  of 
operator  performance  should  therefore  address  both  the  body  and  the  brain.  This  paper  focuses  on  the 
latter.  We  provide  an  extended  list  of  areas  where  neuroscientific  knowledge  may  be  important  like 
training  and  mental  healthcare.  We  zoom-in  on  the  relevance  of  neuroergonomics  and  Brain  Machine 
Interfaces  (BMIs)  and  present  recent  data  from  our  lab.  Up  till  today,  the  majority  of  applied 
neuroscience  research  is  aimed  at  assisting  people  with  medical  limitations,  arid  not  at  performance 
enhancement  for  healthy  users.  Knowledge  transfer  from  patient  orientated  applications  to  military  brain 
performance  enhancement  offers  major  opportunities,  for  example  in  the  design  and  evaluation  of  new 
systems.  We  foresee  that  the  first  applications  of  BMIs  will  be  available  for  workstation  operators  in  high- 
risk  environments.  Future  research  should  be  focussed  on  three  transitions:  1)  from  clinical  and  patient 
apparatus  to  applications  and  equipment  for  healthy  users,  2)  from  lab  (or  controlled )  environments  to  the 
field,  and  3)  from  fundamen  tal  knowledge  to  operational  applications. 
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1.0  INTRODUCTION 

Traditionally,  human  performance  enhancement  focuses  on  the  improvement  of  the  physical  performance, 
and  not  so  much  on  the  cognitive  abilities  of  the  human.  However,  in  current  military  operations  cognitive 
tasks  such  as  information  gathering  and  interpretation,  decision  making,  communication  with  civilians 
(with  or  without  (remote)  language  support),  and  operation  of  complex  equipment  are  required  throughout 
all  ranks  and  levels.  Apart  from  physical  fitness,  cognitive  fitness  has  become  a  prerequisite  to  complete 
mission  essential  tasks.  This  makes  enhancing  cognitive  or  brain  performance  a  relevant  issue. 

Apart  from  mission  essential  tasks,  cognitive  performance  is  also  critical  in  the  interaction  with  systems. 
In  this  respect,  understanding  the  cognitive  state  and  capabilities  of  the  individual  soldier  becomes 
increasingly  important  to  build  the  required  'intelligent'  systems  (see  Figure  1).  There  are  large  individual 
differences  and  these  should  be  taken  into  account  in  operational  deployment  and  the  design  and  use  of 
operational  systems.  In  the  last  decade,  the  contribution  of  neuroscience  has  grown,  based  on  the 
conviction  that  the  human  brain  is  the  basis  for  both  cognitive  tasks  (perception,  planning,  decision 
making)  and  for  physical  performance  (grasping,  aiming,  pointing).  Since  the  brain  also  controls  processes 
throughout  the  body,  and  our  body  interacts  in  turn  with  our  brain,  brain  performance  is  also  relevant  with 
respect  to  physical  performance.  Important  steps  forward  have  been  made  in  fundamental  neuroscientific 
knowledge.  Knowledge  on  the  structure  and  functioning  of  the  brain  grows  every  day. 


Figure  1 :  Examples  of  humanization  of  the  interface;  intelligent  systems  (“virtual  assistants”) 
that  look  and  behave  like  human  beings,  and  that  are  required  to  have  a  better  understanding  of 
the  human  partner's  cognitive  and  emotional  state  and  capabilities. 


From  the  perspective  of  biology,  brain  performance  enhancement  through  pharmacological  and  later  on 
possibly  through  genetic/stem-cell  measures  will  significantly  improve  neurocognitive  abilities.  Today  we 
see  a  lot  of  activities  directed  to  optimise  the  biochemistry  of  the  brain  and  biochemical  signalling.  For 
example,  the  well-described  mental  state  of  flow  with  all  its  positive  effects  seems  to  be  connected  with 
the  activity  of  the  substance  Anandamide,  probably  the  transport  vehicle  for  endorphins  through  the  blood- 
brain  barrier.  Most  of  these  activities  arouse  from  observing  unexpected  side  effects  of  medication  to 
patients  with  psychiatric  diseases,  but  only  are  enabled  by  major  advances  in  understanding  human 
biochemistry.  This  paper  however,  does  not  sketch  the  biological  perspective  but  follows  a  Human  Factors 
Engineering  approach  (see  [1]  in  this  issue  for  a  complementary  view).  It  consists  of  several  parts.  First, 
we  reflect  on  the  potential  contributions  of  neuroscientific  knowledge  to  the  military  domain  and  in 
particular  on  the  areas  neuroergonomics  and  Brain-Machine  Interfaces  (BMI).  We  zoom-in  on 
neuroergonomics  and  BMI  applications  and  report  the  latest  data  from  our  lab  on  the  use  of  brain  signals 
to  improve  human-system  symbiosis.  Finally,  we  discuss  the  way  forward  and  the  (specific)  challenges 
and  opportunities. 
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Figure  2:  Example  of  the  risk  of  perceptual  and  cognitive  overload. 


2.0  MILITARY  RELEVANCE  OF  NEUROSCIENTIFIC  KNOWLEDGE 

Neuroscientific  knowledge  can  potentially  contribute  to  several  military  issues.  In  some  areas,  this 
contribution  will  be  minor,  but  in  others  it  may  actually  be  essential  to  make  progress  towards  human- 
system  symbiosis  [2].  We  give  the  following  list  as  an  indication  of  the  scope  rather  than  as  a  complete 
overview.  We  will  discuss  the  first  four  in  more  detail  in  the  next  sections. 

•  System  design.  Neuroscientific  knowledge  can  have  an  important  contribution  in  the  design 
phase  of  military  systems  (as  far  as  user  cognition  is  relevant  for  the  system  at  hand). 
Traditionally,  the  cognitive  sciences  have  made  major  contributions  to  this  area,  mainly  by 
providing  models  of  functions  like  perception,  memory,  and  motor  skills  that  help  optimise 
system  design.  Example  include  design  considerations  such  as  colour  use,  symbol  size,  and  menu 
structures.  Neuroscientific  knowledge  on  user  aspects  such  as  attention,  engagement,  and 
workload  can  expand  the  set  of  existing  system  design  guidelines. 

•  System  evaluation.  Besides  improving  system  design,  neuroscientific  research  tools  (e.g.,  neuro¬ 
imaging)  can  also  be  used  to  evaluate  systems  by  testing  the  extent  to  which  various  known  brain 
areas  and  signals  are  involved  while  the  system  is  being  used.  Neuro-imaging  may  therefore  help 
determine  whether  and  to  what  extent  systems  stimulate  the  relevant  brain  regions  optimally 
involved  in  a  particular  task.  The  advantage  of  neuroscientific  measuring  methods  is  that  they 
help  us  understand  variables  that  cannot  be  determined  by  behavioural  measurement. 

•  Direct  communication.  People  working  in  challenging  environments  and  at  the  edge  of  their 
physical  and  cognitive  abilities  such  as  military  operators  are  a  major  thrust  for  ergonomic 
innovation  (see  Figure  2).  Developments  in  human  system  integration  have  proven  to  be  of  great 
importance  in  supporting  mission  essential  tasks  in  high-risk  environments.  In  situations  where  an 
operator  has  literally  a  shortage  of  eyes  and  hands,  direct  communication  between  operator  and 
system  without  the  user’s  sensory  or  motor  system  but  based  on  brain  signals,  can  be  considered 
as  an  additional  communication  channel  between  user  and  system. 1 


1  Please  note  that  communication  can  also  be  from  system  to  user,  for  instance  cochlear  implants  for  the  hearing  impaired  and 
electrodes  for  deep  brain  stimulation  for  people  suffering  from  tremors. 
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•  System  adaptation.  Sensory  or  cognitive  overload  is  an  eminent  risk  for  operators  in  information 
rich  environments.  One  of  the  solutions  to  this  risk  is  to  implement  a  system  that  adjusts  the 
information  presentation  (e.g.,  postpone  messages)  or  the  task  allocation  (e.g.,  take  over  tasks 
from  the  user)  to  the  user’s  current  capabilities.  To  accomplish  this,  such  an  adaptive  system 
needs  information  on  the  present  state  of  the  user,  for  example  on  visual  attention,  task 
engagement,  or  cognitive  load.  Brain  signals  may  be  valuable  indicators  of  operator  state,  adding 
to  existing  physiological  indicators  such  as  heart-rate  variability.  Another  possibility  is  that  brain 
signals  may  indicate  rare  but  threatening  situations  such  as  pilot  spatial  disorientation2.  An 
advantage  of  neural  measures  is  that  they  can  add  vital  information  to  sometimes  less  reliable 
behavioural  measures.  For  example,  we  can  use  eye  movement  technology  to  determine  what 
someone  is  looking  at,  but  not  whether  that  person  is  actually  perceiving  the  visual  information  or 
attending  to  music  playing  on  thee  radio  or  planning  an  action.  By  measuring  brain  signals  in  the 
visual  cortex,  we  may  be  able  to  determine  the  focus  of  someone’s  attention.  A  step  beyond 
identifying  the  locus  of  attention  is  distilling  information  about  the  objects  an  operator  has  seen. 
Recently,  neurocognitive  researchers  have  been  able  to  accurately  identify  which  of  two  objects  a 
respondent  had  been  looking  at,  for  instance  a  horse  or  a  cow,  a  bird  or  a  butterfly,  based  on 
twenty  seconds  of  fMRI  data  (for  an  overview  see  [3]). 

•  Personnel  selection.  Neural  indicators  may  help  to  assess  and  predict  the  extend  to  which  an 
individual  is  able  to  cope  with  stressful  environments,  high  information  loads,  etc. 

•  Training.  Most  aspects  of  training  are  related  to  the  brain  and  its  plasticity.  Measuring  this 
plasticity  and  changes  in  the  brain  can  help  to  improve  training  methods  in  general  and  an 
individual’s  training  schedule  in  particular.  With  respect  to  the  latter,  indicators  such  as  learning 
state  and  rate  of  progress  (novice  /  expert)  are  useful.  To  sketch  a  picture  of  recent  neuroscientific 
progress:  Neuroscientists  showed  that  they  were  able  to  predict  whether  or  not  a  respondent  would 
remember  a  specific  word  based  on  fMRI  patterns  only.  A  topic  gaining  relevance  to  military 
performance  is  the  emerging  scientific  field  of  neuroplasticity.  The  term  refers  to  the  brain's 
ability  to  change  its  physiological  structure  and  function,  in  particular  by  expanding  or 
strengthening  circuits  that  are  used,  and  by  shrinking  or  weakening  those  that  are  rarely  engaged. 
The  science  of  neuroplasticity  has  mostly  documented  brain  changes  that  reflect  physical 
experience  and  input  from  the  outside  world.  Recently  also  the  change  in  response  to  purely 
internal,  mental  signals  has  been  investigated  by  EEG  [4]  and  fMRI  by  probing  Buddhist  long 
term  meditators.  Early  work  from  the  1970s  and  1980s  probing  Transcendental  Meditation 
practitioners  already  revealed  dramatic  increase  in  synchronicity  and  activity  of  brain  waves 
through  EEG  measurements,  but  could  not  correlate  it  to  physiological  changes  in  the  brain.  These 
findings  allow  for  new  types  of  mental  training  to  improve  or  even  aquire  mission  critical 
abilities. 

•  Resilience  and  mental  healthcare.  Neuroscientific  knowledge  could  also  help  to  protect  military 
personnel  against  brain-related  stressors  (resulting  in  a  better  preparation  for  deployment),  or  help 
to  recovery  after  stressful  experiences.  Neurofeedback  can  be  considered  as  direct  communication 
between  brain  and  system.  However,  the  aim  here  is  not  to  communicate  messages  or  to  control  a 
machine,  but  to  train  producing  (or  suppressing)  brain  signals  themselves  in  order  to  improve 
well-being.  Neurofeedback  is  already  considered  a  valuable  tool  in  treating  for  instance 
depression,  sleep  disorders,  and  post  traumatic  stress  disorder. 

•  Cognitive  performance.  Neurofeedback  and  other  neuroscientific  techniques  can  also  be 
employed  to  increase  cognitive  performance,  concentration  and  mental  fitness  in  general. 


2 

“  Please  note  that  this  is  a  hypothetical  example.  As  far  as  we  know,  no  brain  signal  correlates  of  Spatial  Disorientation  have 
been  identified  yet. 
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•  Information  operation.  A  niche  application  with  military  relevance  is  to  employ  the  superior 
pattern  recognition  capacities  of  the  human  brain  to  analyse  large  amounts  of  visual  imagery  like 
satellite  images  at  high  speed  and  select  relevant  images  or  detect  changes  over  time. 

•  Rehabilitation.  After  deployment,  neuroscientific  knowledge  can  also  contribute  to  physical  (and 
mental)  healthcare,  e.g.  through  knowledge  on  neural  recovery,  to  support  people  with  reduced 
sensory,  motor,  or  cognitive  functions  and  to  treat  mental  disorders.  The  use  of  BMIs  in  motor 
revalidation  is  currently  investigated  by  for  instance  the  EU  project  TOB1  (Tools  for  brain- 
computer  interaction).  The  idea  is  to  give  feedback  to  the  responses  of  motor  neurons  as  to  keep 
them  active  even  though  the  body  is  (currently)  not  responding. 

Of  the  list  above,  we  will  zoom-in  on  the  first  four  bullets:  system  design,  system  evaluation,  direct 
communication,  and  system  adaptation.  The  first  two  of  these  are  also  known  under  the  designation, 
neuroergonomics,  while  the  latter  two  are  considered  as  Brain  Machine  interfaces  (BMI)  or  Brain 
Computer  interfaces  (BCI). 

2.1  Neuroergonomics:  system  design  and  evaluation 

Recently,  more  and  more  automation  has  been  made  available  to  the  human.  However,  a  possible 
downside  can  be  the  increasing  information  complexity  and  volume,  and  an  increasing  autonomy  of 
technical  systems  with  out-of-the-loop,  situational  awareness  and  vigilance  problems  as  a  result.  To  cope 
with  these  issues,  human  factors  engineering  is  essential.  We  should  keep  the  human  as  central  focal  point 
of  the  design  process,  to  prevent  human  error  and  major  consequences  after  system  implementation. 
Neuroergonomics  is  an  emerging  field  that  combines  neuroscience  with  human-system  interaction  studies 
in  order  to  evaluate  how  well  a  technology  matches  human  capabilities  and  limitations.  The  term 
neuroergonomics  was  first  coined  by  Raja  Parasuraman  [5].  Parasuraman  explains  the  aim  of  the  field  in 
the  following  words:  “to  use  the  discoveries  of  human  brain  and  physiological  functioning  both  to  inform 
the  design  of  technologies  in  the  workplace  and  home,  and  to  provide  new  training  methods  that  enhance 
performance,  expand  capabilities,  and  optimize  the  fit  between  people  and  technology”.  An  illustrative 
example  is  the  recent  research  on  cell  phone  use  during  driving  [6,  7,  8]  and  brain-imaging  results  that 
indicates  that  even  the  hands-free  or  voice  activated  use  of  a  mobile  phone  is  as  dangerous  as  being  under 
the  influence  of  alcohol  during  driving.  The  listening-and-drive  mode  produced  a  37%  decrease  in  activity 
in  the  parietal  lobe,  which  is  associated  spatial  processing,  critical  for  navigation.  Activity  also  decreased 
in  the  occipital  lobe,  which  processes  visual  information  [9] . 

2.2  Brain  Machine  Interfaces:  direct  communication  and  system  adaptation 

Brain  Machine  Interfaces  (BMIs)  enable  direct  communication  between  the  user’s  brain  or  nervous  system 
and  a  machine  without  involving  the  motor  system.  BMIs  are  an  embryonic  technology  but  remarkable 
accomplishments  have  been  reported  in  the  last  decade,  for  example  in  the  ability  to  use  single  trial  EEG 
measurements  [10]  and  apply  BMIs  in  fast  control  processes  [1 1], 

BMIs  can  be  roughly  divided  in  active,  reactive  and  passive  BMIs  [12].  With  active  or  reactive  BMIs, 
users  consciously  operate  a  machine  handsfree.  In  active  BMIs,  they  do  this  for  example  by  imagining  left 
versus  right  hand  movements  which  the  computer  then  can  decode  as  the  desired  direction  of  a  cursor  on  a 
screen  or  of  a  vehicle3.  In  reactive  BMIs,  users  are  presented  with  different  kinds  of  so-called  probe 
stimuli  (e.g.  images  or  differently  pitched  tones)  that  elicit  certain  detectable  brain  signals  whenever 
attention  is  consciously  focused  on  them.  As  in  active  BMIs,  reactive  BMIs  allow  the  communication  of 
messages  without  the  use  of  the  motor  system.  An  advantage  of  active  over  reactive  is  the  independence  of 
probe  stimuli;  an  advantage  of  reactive  over  active  is  the  speed  of  communication  due  to  the  relatively 
clear  brain  signals  even  without  training  of  the  user.  A  special  application  of  reactive  BMIs  lies  in  image 


To  date,  motor  imagery  has  been  the  most  commonly  used  task.  However,  eventually  other  cognitive  tasks  may  prove  to  be 
more  effective  [203], 
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classification  [13].  Since  the  (human)  brain  is  still  notably  better  in  certain  kind  of  image  recognition  or 
classification  tasks  than  computer  algorithms,  one  can  make  use  of  this  capability  by  presenting  the  brain 
with  large  sets  of  very  quickly  presented  images  where  the  human  has  to  pay  attention  to  a  certain  target. 
Instead  of,  or  in  addition  to,  pressing  a  button,  brain  signals  can  be  used  to  speed  up  the  classification  task. 
The  third  kind  of  BMIs  is  called  passive.  Passive  BMIs  are  not  intended  to  control  machines  but  to 
enhance  human-system  interaction  and/or  inform  the  user  about  his/her  state.  These  BMIs  do  not  require 
mental  effort  or  consciously  generated  brain  signals,  but  use  spontaneously,  'automatically'  generated 
signals  that  arise  when  doing  a  certain  task.  Passive  BMIs  can  monitor  aspects  such  as  (visual)  attention, 
engagement,  drowsiness  or  mental  workload  and  initiate  proper  action  such  as  issue  a  warning,  adapt  the 
information  presentation  or  allocate  tasks  differently  between  user  and  system.  Also,  errors  can  be 
recognized  and  corrected  [14]. 

The  ultimate  goal  is  the  use  of  these  passive  BMIs  to  create  a  symbiotic  relation  between  human  and 
machine  in  a  closed-loop  system,  which  continually  optimizes  its  performance.  Extended  research  has 
already  been  done  in  the  field  of  Augmented  Cognition,  which  aims  to  compensate  for  temporal 
limitations  in  human  information  processing  [15]. 


3.0  BMI  DEVELOPMENTS  AT  TNO 

In  this  section,  we  will  discuss  a  reactive  BMI  and  two  kinds  of  passive  BMIs  (one  to  monitor  errors  and 
one  to  monitor  workload)  from  our  lab. 

3.1  Handsfree  navigation 

The  traditional  application  domain  for  handsfree  navigation  BMIs  is  for  severally  disabled  people  who 
lack  the  motor  capabilities  to  communicate.  In  this  domain,  active  BMIs  are  the  principal  choice  because 
users  can  devote  large  parts  of  their  cognitive  resources  to  the  navigation  task.  For  military  applications, 
the  advantage  of  handsfree  navigation  is  evident,  but  operators  will  certainly  not  be  in  a  situation  where 
they  can  easily  devote  cognitive  resources  to  the  navigation  task  only.  For  example,  brain-based 
applications  may  be  introduced  to  navigate  a  remotely  operated  vehicle  more  intuitively  than  via 
complicated  control  devices  while  the  user  is  engaged  in  cognitively  demanding  tasks  such  as  surveillance 
or  ordnance  disposal.  Thus  we  should  look  for  interfaces  that  benefit  from  the  handsfree  aspect  but  also 
require  acceptable  cognitive  effort.  Therefore,  our  principal  choice  is  for  reactive  or  passive  BMIs. 
Ultimately,  a  BMI  should  be  able  to  recognize  the  brain’s  desire  to  navigate  somewhere,  so  that  the  user 
does  not  have  to  plan  and  execute  the  appropriate  motor  commands  to  control  a  complex  interface. 

Many  reactive  BMIs  are  based  on  Event  Related  Potentials  (ERPs),  especially  the  P300  ERP.  The  P300  is 
a  positive  peak  in  EEG  that  occurs  approximately  300  ms  after  presentation  of  an  attended  'target' 
stimulus,  presented  within  a  sequence  of  other  ignored  'distracter'  stimuli.  Because  users  can  choose  for 
themselves  which  of  the  presented  stimuli  is  the  target  to  attend  to,  and  because  the  P300  can  relatively 
easily  be  detected,  this  signal  has  often  been  used  to  drive  BMIs.  A  well-known  example  is  the  P300 
speller  in  which  rows  and  columns  of  a  matrix  consisting  of  letters  are  sequentially  flashed  in  random 
order  [16].  Every  time  the  row  or  column  is  flashed  that  contains  the  symbol  that  the  user  wants  to  spell,  a 
P300  occurs.  In  this  way,  users  can  spell  directly  with  their  brain.  Other  P300  based  BCIs  have  been 
designed  to  control  a  robot  wheelchair  [17],  to  switch  devices  on  or  off  in  a  virtual  room  [18]  and  to  stop  a 
virtual  car  [19]. 

As  far  as  we  know,  all  P300  BMIs  up  to  date  have  used  visual  or  auditory  stimuli  to  elicit  P300s. 
However,  for  military  purposes  tactile  stimuli  can  be  an  interesting  alternative.  These  stimuli  can  be 
delivered  by  tactors  that  are  hidden  under  the  user's  clothes,  making  the  ‘stealthy’.  Using  tactile  stimuli 
will  also  keep  the  eyes  and  ears  of  the  user  free,  and  these  are  often  primarily  required  to  ensure  the 
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soldier’s  performance  and  safety.  Tactors  applied  around  the  waist  have  proven  to  be  successful  as 
navigation  display  [20],  and  may  therefore  be  a  natural  choice  for  designing  an  intuitive  BMI  for 
navigation. 

In  a  series  of  experiments,  we  asked  participants  to  focus  on  one  of  several  tactors  around  the  waist 
(Figure  3)  that  vibrated  in  random  order  with  or  without  corresponding  visual  stimuli  on  a  monitor.  The 
idea  is  that  in  this  way  they  can  communicate  a  desired  direction  of  movement:  a  P300  occurring  after  the 
tactor  on  the  navel  vibrated  means  'I  want  to  go  forward'  and  after  the  tactor  on  the  left  side  'I  want  to  go 
left'.  In  the  first  experiment,  we  demonstrated  that  these  kind  of  stimuli  indeed  elicited  P300s  and  that  they 
can  be  at  least  as  strong  as  visual  P300s  [21].  Off-line  classification  showed  that  a  classification  model  can 
successfully  pick  out  the  target  direction  from  the  distracters  after  only  one  presentation  of  each,  though 
performance  increases  with  more  presentations  (Figure  4).  Also,  we  found  that  presenting  stimuli 
bimodally  (in  the  tactile  and  visual  modality)  increases  the  performance  of  the  BMI.  This  indicates  that 
tactile  stimuli  are  also  powerful  enough  to  add  to  the  traditional  visual  BMIs.  We  suspect  that  this 
advantage  may  be  even  larger  when  the  visual  and  tactile  stimuli  are  more  in  congruence  than  in  our 
experiment  (i.e.,  perceived  at  the  same  location  in  space  rather  than  on  the  waist  and  on  a  screen).  In 
subsequent  experiments,  we  tested  several  spatial  and  temporal  parameters  of  a  tactile  BMI  where 
participants  received  online  feedback.  Findings  include  that  from  a  P300  point  of  view,  two,  four  and  six 
tactors  are  equally  suitable  [22]  and  that  the  time  between  the  start  of  the  stimuli  could  be  at  almost  twice 
as  short  in  comparison  to  the  experiments  before  (376  ms  instead  of  626  ms)  without  loosing  any 
classification  accuracy. 


Figure  3:  Participant  showing  the  tactile  vest  and  EEG  cap  used  in  the  experiment. 
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Figure  ???:  Classification  performance  (percentage  correct)  as  a  function  of  stimulus  modality 
(tactile,  visual,  bimodal)  and  the  number  of  stimulus  presentations  (1-10).  The  grey  line  indicates 

chance  performance. 


3.2  Error  correction 

Although  the  cognitive  mechanisms  underlying  human  error  detection  processes  are  not  yet  fully 
understood,  various  EEG  studies  [23,  24,  25]  have  shown  that  human  error  detection  is  associated  with 
typical  patterns  in  the  EEG  signal,  so-called  error  potentials.  In  general,  these  error  potentials  can  be 
observed  whenever  the  actual  outcome  of  an  action  does  not  match  the  intended  outcome.  The  exact 
structure  of  error  potentials  depends  on  the  type  of  error.  On  the  one  hand,  the  machine  may  commit  an 
error  (for  example  when  wrongly  interpreting  a  user  intention  due  to  restricted  information);  on  the  other 
hand,  errors  may  occur  due  to  erroneous  behaviour  of  the  user  (for  example  when  accidentally  pressing  a 
wrong  button).  We  call  these  machine  errors  and  self -generated  errors,  respectively.  Error  potentials 
provide  us  with  information  that  is  not  directly  observable  from  the  'outside'  and  could  be  used  by  BMIs  to 
quickly  detect  and  possibly  correct  errors  without  disturbing  and  requiring  any  effort  of  the  user.  Some 
studies  showed  that  BMIs  can  indeed  detect  single  machine  errors  [12]  and  self -generated  errors  [26]. 

As  alluded  to  in  the  previous  section,  tactile  interfaces  started  to  attract  the  attention  in  the  field  of  human- 
machine  interaction.  Van  Erp  [20]  pointed  out  various  advantages  of  tactile  interfaces.  These  include  their 
potential  to  lower  cognitive  workload  in  other  modalities  and  their  ability  to  intuitively  direct  a  user’s 
attention  (a  proverbial  tap-on-the-shoulder).  Because  of  the  increasing  importance  of  tactile  interfaces  in 
human-machine  systems,  it  is  of  interest  to  determine  whether  error  potentials  similar  to  the  ones  found  in 
the  visual  and  auditory  domain  are  observable  for  tactile  stimuli.  There  exists  only  one  study  [27]  that 
indicates  that  self  generated  error  potentials  can  be  elicited  in  the  tactile  domain.  We  explored  the  EEG 
patterns  related  to  error  processing  in  the  tactile  modality  for  both,  self-generated  and  machine  errors  [28]. 

Subjects  were  asked  to  move  a  tactile  cursor  to  a  tactile  target.  Cursor  and  target  were  presented  using  the 
tactile  display  as  described  in  the  previous  section.  The  target  was  continuously  vibrating  with  on-  and  off 
times  of  100  ms.  The  cursor  was  presented  for  400  ms.  Subsequently,  subjects  were  presented  with  a 
visual  cue  indicating  a  movement  direction  (clockwise  or  counter  clockwise).  By  pressing  an  'accept' 
button  or  a  'reject'  button  they  could  either  make  the  cursor  move  in  the  desired  direction  or  not.  In  some 
conditions,  the  machine  committed  an  error  by  moving  the  opposite  direction  of  that  indicated  by  the  user. 
In  other  conditions,  the  visual  cues  were  made  more  difficult  to  interpret,  such  that  the  users  made 
relatively  many  errors.  Figure  5  shows  the  averaged  error  potentials  as  found  in  this  study.  Interestingly, 
for  self-generated  errors,  EEG  traces  between  correct  and  erroneous  trials  already  start  to  differ  before  the 
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button  has  been  pressed,  indicating  that  the  brain  is  aware  it  has  initiated  an  erroneous  movement  before  it 
is  executed.  This  could  potentially  be  very  valuable  information  to  be  used  by  a  BMI.  Mean  classification 
rate  for  both  types  of  errors  was  around  70%,  albeit  with  relatively  many  false  alarms.  This  can  most 
probably  be  improved  by  providing  the  classification  model  with  more  training  data. 

Computer  error 


Tactor  \  Error 


Own  Error 


Button  push 

Difference 


-200  -100  0  100  200  300  400  500  600  700  800 


Figure  5:  Tactile  error  related  EEG  signals. 


3.3  Operator  State  and  adaptive  interfaces 

The  ability  to  continuously  monitor  workload  in  a  real-world  environment  would  have  important 
implications  for  the  design  of  human  machine  interfaces  as  well  as  the  real-time  online  improvement  of 
interaction  between  humans  and  machines.  For  example,  based  on  an  online  measure  of  operator  state,  one 
could  adjust  the  task  allocation  between  operator  and  machine  during  high  workload,  let  the  automatic 
pilot  take  over  control  during  pilot  spatial  disorientation,  or  send  a  baggage  inspector  for  a  break  when 
their  visual  attention  is  too  low.  Several  studies  showed  that  different  frequency  bands  in  the  EEG  power 
spectrum  vary  with  workload  [29,  30,  31],  fatigue  [32,  33,  34],  or  visual  attention  [35,  36]. 

NIRS  (Near  Infrared  Spectroscopy)  is  a  less  investigated  brain  imaging  technique  that  measures  the 
relative  concentration  of  oxyhemoglobin  and  deoxyhemoglobin  within  the  cortex.  There  are  indications 
that  workload  affects  NIRS  signals  as  well  [37,  38].  Since  EEG  and  NIRS  measure  different  physiological 
signals  and  are  susceptible  to  different  noise  sources,  it  is  possible  that  a  combination  of  both  techniques 
will  enhance  our  ability  to  distinguish  between  different  levels  of  workload  online.  We  investigated  the 
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effect  of  different  workload  levels  on  EEG  and  NIRS,  and  evaluated  whether  using  both  signals  in  a 
classification  model  will  improve  offline  workload  identification. 

Workload  was  varied  using  the  n-back  task  [39].  Ten  subjects  viewed  a  screen  on  which  letters  were 
presented  successively.  Their  task  was  to  indicate  by  button  press  whether  or  not  the  current  letter  was  the 
same  as  the  one  presented  just  before  (1-back)  or  the  one  before  that  (2-back).  In  the  0-back  condition 
subjects  judged  whether  the  letter  is  an  Y  or  not.  EEG  and  NIRS  signals  were  recorded  simultaneously 
(Figure  6). 


Figure  6:  Participant  wearing  a  cap  that  contains  both  EEG  and  NIRS  sensors.  The  three  NIRS 

sensors  are  indicated  by  the  three  ovals. 


Preliminary  analyses  (see  also  Figure  7)  show  that  the  EEG  recordings  generally  resulted  in  good 
classification  between  the  2-back  and  both  the  1-back  (70.8%)  and  the  0-back  (73.3%)  conditions,  but 
NIRS  not  (53.5%  and  61.3%,  respectively).  Also,  combining  EEG  and  NIRS  (56.5%  and  61.1%, 
respectively)  did  not  improve  the  classification  results  compared  to  EEG  alone.  The  limited  performance 
of  NIRS  as  workload  indicator  may  be  due  to  the  fact  that  we  have  not  yet  developed  classifiers 
specifically  for  NIRS  signals  but  used  EEG  classifiers  for  the  both  the  EEG  and  the  NIRS  data.  Further 
analyses  should  reveal  whether  NIRS  signals  can  add  to  EEG  in  classifying  levels  of  workload. 
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0-back  1-back  2-back 


Figure  7:  Example  result  for  one  participant  in  the  workload  study.  Effect  of  task  difficulty  on  the 
EEG  power  spectrum  is  given  in  panel  a)  and  on  the  oxyhemoglobin  level  in  panel  b). 


4.0  CONCLUSIONS 

We  argued  that  current  military  operations  not  only  challenge  the  physical  capabilities  of  soldiers,  but  also 
their  cognitive  abilities.  Performance  enhancement  should  therefore  also  be  focussed  on  cognitive 
performance  and  thus  the  brain.  Or  as  Parasurman  and  Wilson  [40]  put  it:  to  examine  behaviour  and  the 
mind  at  work,  one  should  study  what  makes  work  possible  -  the  brain.  The  scientific  discipline  that 
studies  the  brain  is  neuroscience,  and  hence  we  should  identify  neuroscientific  knowledge  that  is  of 
relevance  to  the  military  domain.  We  have  given  an  extended  list  of  the  possible  neuroscientific 
contributions  and  zoomed-in  in  more  detail  to  neuroergonomics  and  BMIs.  Until  today,  the  majority  of 
applied  neuroscience  research  is  aimed  at  assisting  people  who  suffer  from  a  physical,  perceptual  or 
cognitive  challenge  and  not  at  performance  enhancement  for  healthy  users.  This  situation  opens  up 
opportunities  for  spin-off  and  spin-in  between  advanced  (military)  Human  System  Interaction  knowledge 
and  the  accomplishments  in  neurotechnology  for  patients.  Currently,  there  is  inadequate  structural 
knowledge  development4  for  military  specific  fields  while  fundamental  knowledge  has  limited  value  for 
applications,  especially  in  the  military  domain.  Several  new  challenges  arise:  how  to  build  an  application 
that  is  fast,  accurate,  easy  to  use,  and  that  can  be  operated  in  strenuous  environments  and  based  on  sensor 
systems  (EEG,  NIRS,  or  others)  that  are  preferably  portable,  easy  to  don,  and  non-obtrusive.  Also,  low 
bitrates  may  be  acceptable  for  patient  applications,  but  may  be  of  less  interest  to  healthy  users  [41].  These 
issues  add  to  challenges  that  have  to  be  dealt  with  in  applications  for  patients  such  as  large  differences 
between  individuals  and  even  BCI  illiteracy.  From  a  military  perspective,  knowledge  development  should 


4  Several  NATO  countries  have  started  research  programs  or  feasibility  studies  into  this  matter,  amongst  others  the  UK,  the  US, 
Canada,  Germany  and  the  Netherlands. 


RTO-MP-HFM-181 


32  - 11 


Brain  Performance  Enhancement  for  Military  Operators 


focus  on  three  transitions:  1)  from  clinical  and  patient  applications  to  applications  for  healthy  users,  2) 
from  lab  (or  controlled)  environments  to  the  field,  and  3)  from  fundamental  knowledge  to  operational 
applications.  These  transitions  will  also  involve  ethical  issues.  Everyone  will  agree  that  personality,  free 
will  and  mental  abilities  are  interwoven  with  brain  function,  which  become  more  important  if  we  shift 
from  therapeutic  BMIs  towards  enhancement  BMIs  (see  Figure  8).  Therapeutic  BMIs  focus  on  people 
with  degraded  functioning  (for  example  as  a  result  of  a  stroke),  while  enhancement  BMIs  focus  on 
expanding  the  capacities  of  normal  functioning  individuals  which  is  an  important  difference  from  an 
ethical  point  of  view,  especially  when  medical  risks  are  involved  as  with  invasive  sensor  systems. 

In  conclusion 

•  Neuroscientific  knowledge  can  contribute  to  the  military  domain  in  designing  and  evaluating 
equipment  (neuroergonomics),  selection  and  training  of  personnel,  increasing  resilience  and  in 
user  system  communication  (BMI). 

•  Recent  developments  in  neuroergonomics  are  concerned  with  the  neural  basis  of  cognitive 
functions  (such  as  perception  and  decision  making)  and  of  physical  performance  (such  as  grasping 
and  aiming).  These  developments  are  essential  new  ways  to  cope  with  increasing  problems  in 
human-system  integration. 

•  BMIs  carry  with  them  the  expectation  of  the  future,  they  are  among  us  and  are  here  to  stay.  Tens 
of  thousands  of  individuals  already  use  a  BMI  for  different  therapeutic  applications.  Still,  these 
BMIs  are  one-way  tools  to  either  actively  send  signals  from  the  brain  to  a  technical  system  or  v.v. 
(including  cochlear  implants  and  deep  brain  stimulation  for  people  suffering  from  tremors). 
Especially  non-invasive  BMIs  are  not  yet  capable  of  transferring  information-bearing  signals  into 
the  brain.  Thus,  pure  interactive  brain-machine  communication  is  not  viable  yet5. 

•  Generally  speaking,  one  could  say  that  as  BMI  qualities  improve,  the  application  of  BMIs  will  no 
longer  be  limited  to  patients.  Instead,  the  field  may  gain  momentum  at  the  moment  the  military, 
gaming,  entertainment,  and  other  industries  get  fully  involved  and  BMIs  will  become  available  to 
an  exponentially  increasing  set  of  patients  and  non-patient  groups. 

•  We  foresee  that  the  first  applications  of  BMIs  will  be  passive  systems  for  workstation  operators  in 
high-risk  environments.  Active  systems  still  are  still  too  slow  in  terms  of  precise  intentional 
information  transfer.  In  recent  years,  major  steps  have  been  accomplished  in  passive  BMIs  in  lab 
environments  for  instance  to  implement  adaptive  automation.  Also,  applications  for  workstation 
operators  do  not  depend  on  the  development  of  wearable  (sensor)  systems. 

•  The  thrust  from  technology  with  enablers  like  advanced  materials,  nanotechnology, 
microintegration  etc.  will  open  up  possibilities  to  contact  the  brain  in  a  more  effective  and  gentle 
way,  therewith  paving  the  way  for  broader  application  of  neurotechnology  in  the  military  domain. 


For  healthy  users,  one  could  consider  to  use  the  peripheral  nervous  system  to  establish  communication  from  system  to  the 
brain  therewith  bypassing  the  human  senses  with  their  shortcomings  but  without  needing  to  contact  the  brain  directly. 
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Figure  8:  Trends  over  the  past  few  decades  clearly  show  that  technology  and  systems  are 
coming  closer  to  the  human  body  and  are  nowadays  often  worn  on  the  body.  Next  generation 
interfaces  for  healthy  users  may  be  partly  inside  the  body.  This  step  has  already  been  taken  for 
special  patient  groups.  Although  ethical  questions  will  certainly  play  a  role  in  this  process, 
applications  for  healthy  users  are  based  on  implanted  chips  are  on  the  market,  like  VeriChip 
(Delray  Beach,  FL)  implants  to  pay  for  drinks  in  a  discotheques  (picture  from  Baja  Beach  Club, 

Rotterdam,  The  Netherlands;  www.baia.nh. 
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